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there has seemed to be no economical, valid, and reliable alternative. This 
paper reports on an alternative alignment process for evaluating quality 
teaching and learning that fits the requirements of economy, validity, and 
reliability, and offers additional institutional, faculty, and student 
benefits. The method measures both the individual student's and the 
lecturer's expectations for change in three process objectives that underlie 
quality teaching and learning: skills, understanding, and attitudes. These 
three objectives and their assessment are operationally defined. Methods for 
using these three process objectives as vehicles for teaching, learning, and 
course development are supported by the institution through staff and course 
development programs and through recognition of their assessment in student 
grades and in faculty promotion and tenure decisions. The post-course 
criteria measuring quality teaching and learning are students' high academic 
attainments and course enjoyment. Teaching techniques for attaining these 
goals are left as a matter of informed professional choice to the lecturer. 
The separately measured in-course predictors of these criteria are the 
alignments of student/lecturer expectations of change in the three 
objectives. The theory behind this approach is that quality learning results 
when students and lecturer are working toward the same goals. This paper 
introduced the method and presents evidence validating the theory and showing 
that the indicators correlate with the criteria. (Author/SLD) 
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Abstract 

The research literature on the Student Evaluation of Teaching (SET) is replete with criticisms of 
the process, its applications and the student feedback questionnaires that it uses. Despite these 
multifarious criticisms, SETs continue to be used because there has seemed to be no economic, 
valid and reliable alternative. This paper reports an alternative alignment process for evaluating 
quality teaching and learning that fits these requirements and offers 10 additional institutional, 
faculty and student benefits. The method measures both the individual student’s and the Lecturer’s 
expectation for change in three process objectives that underlie quality teaching and learning: 
Skills, Understanding and Attitudes. These three objectives and their assessment are operationally 
defined. Methods for using these three process objective as vehicles for teaching, learning and 
course development are supported by the institution through staff and course development 
programmes and by recognition of their assessment in student grades and in faculty promotion 
and tenure decisions. 

The post-course criteria measuring quality teaching and learning are students’ high academic 
attainments and course enjoyment. Teaching techniques for attaining these goals are left as a 
matter of informed professional choice to the lecturer. The separately measured in-course 
predictors of these criteria are the alignments of student/lecturer expectations of change in the 
three objectives; the theory being that quality teaching and learning results when students and 
lecturer are working towards the same goals. 

This paper introduces the method and presents evidence validating the theory and showing that 
the indicators correlate with the criteria. 



Introduction 

This paper offers an alternative form of assessing quality teaching in tertiary institutions. The traditional 
method of assessing quality of teaching has been questionnaires that ask students to anonymously rate the 
quality of teaching on a 4 or 5 point Likert scale from strongly disagree to strongly agree. In the literature 
these forms are called Student Evaluations of Teaching (SETs). 

SETs have been used in universities for more than thirty years as part of the Quality Assurance Cycle to 
assess the quality of teaching and as an indicator of successful teaching for promotion and tenure decisions. 
Unfortunately, their use has been accompanied by many counter- productive effects such as discouraging 
innovation, and deterring instructors from challenging students (Damron, 1995; Murray, 1984; Ruskai, 
1996). Although their outcomes are intended to improve teaching, a major negative effect of also using 
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them for promotion and tenure decisions has been to contribute to the lowering of academic standards. In 
the copious literature on the subject, this effect is referred to as ‘grade inflation’ or ‘dumbing down courses’ 
and some universities who use SETs now make statistical adjustments for these effects (Gillmore, & 
Greenwald, 1999). SETs have become known as little more than ‘smile sheets’ measuring popularity and 
‘customer satisfaction’ (Altschuler, 1999), and lecturers have developed many methods for improving 
their SET scores that do not necessary improve teaching (Crumbley, 1995). Its seems that one reason SETs 
continue to be used is that there has not been an expedient alternative. This paper introduces such an 
alternative - an alignment method. 

The Alignment method 

There are many psychometric instruments that use what is referred to here as ‘alignment methods’. In an 
alignment method a respondent’s current state is assessed and his/her ideal state is also assessed on the 
same indicators. The difference between the current and ideal states is the alignment. Where the difference 
is large, there is poor alignment which is indicative of problems. Where the difference is small, alignment 
is good which indicates that the current state is close to the ideal. Improved alignment can also be used as 
a measure of successful intervention strategies. What is crucial to the alignment method is the choice of 
indicators measured to assess the current and ideal states. In tertiary education there are three process 
objectives that are emphasized in quality teaching and learning. 

These are Skills, Understanding and Attitudes operationally defined here as: 

1 . Skills - learning of facts/processes. Assessed by speedy accurate reproduction. 

2. Understanding - professional competence. Assessed by justification of novel application. 

3. Attitudes - professional values. Assessed by demonstration. 



The alignment method uses these as indicators. It should be noted that critical thinking is promoted by 
teaching and assessment of professional competence. This is because there are no right/wrong answers and 
only justifications are assessed. Alignment is not based on the course objectives. Course objectives and 
content are used as vehicles for emphasizing the desired degrees of Skills, Understanding and Attitudes. 
This emphasis will vary according to the course level and culture of the subject taught. 



What are aligned are ‘changes expected by the lecturer’ and ‘changes expected by the students’ in each of 
these three indicators. Numerically stated: Alignment = changes expected by Lecturer - changes expected 
by students. Zero is the perfect score, the theory being that students achieve higher standards if they and 
their lecturer are working towards the same changes. The following Figure 1 shows the seven core questions 
that the lecturer and each student answers for the alignment to be calculated. 



These forms are confidential, not anonymous. In fact, students have to pass a test when they enrol so as to 
earn the right to be considered as informed assessors. Individual’s alignments can be grouped to assess the 
mean alignment of any student sub-group of interest - males v females, experienced v novice students, 
older v younger, option 1 v option 2 students, etc. 

Two alignment scores are calculated; 

• Alignment of Scope (changes in absolute quantity) and 

• Alignment of Proportions (changes in relative quantity) 
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Figure 1 : Five minute assessment form 



As it is now on this course As it should be on this course 



(i) Emphasis on Skills 
(getting it right) 



Your 
estimate 
out of 100 



Your 
estimate 
out of 1 00 



Write a 
Number 
in each 
box 



(ii) Emphasis on 
Understanding 
(knowing why) 



As it is now on this course As it should be on this course 



Your 
estimate 
out of 1 00 



Your 
estimate 
out of 1 00 



Write a 
Number 
in each 
box 



(iii) Emphasis on Attitudes 
(relevance to your life) 



As it is now on this course 

I lYour 

estimate 

lout of 100 



As it should be on this course 



Your 
estimate 
out of 1 00 



Write a 
Number 
in each 
box 



How much so far have 

you enjoyed your experience of 

the teaching on this course? 



Your Write a 

estimate Number 
out of 1 00 in the box 



However, these formative alignment indicators, that are measured during the course, are only predictors of 
quality teaching. They are not the criteria of quality teaching. The two accepted summative criteria of 
quality teaching are: 

• Academic standards and 

• Enjoyment of learning 



Validation of the theory 

When the courses are over and the academic results are compared with the alignment scores, it is possible 
to validate the theory for each course, and for each sub-group of students taking each course by 

correlating the Alignment of Scope with Academic standards and by correlating the Alignment of Proportions 
with Enjoyment of learning. 



Further, when the course has finished it is possible to use sensitivity analyses on the data to calculate those 
lecturer’s changes that would have most aligned the teaching and thus maximized the academic results and 
enjoyment of the students. It is seen from actual alignment data that having chosen these preferred changes 
would have increased the correlations between alignment and academic standards, thus further validating 
the theory that alignments are predictors of quality teaching. This is illustrated by data in Figure 2. 
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Illustrative Results 

Figure 2 illustrates a typical data input, analysis and results sheet for the alignment assessment of a course. 



Figure 2: Example input, analysis and results for the alignment method 



Part 1 For the Lecturer 


Course 


ED40C 


Date 


| 15/11/00 


Name 






Skills 


Understanding 


Attitudes 


Enjoyment 


Variables of Interest 


For Validation: 
Academic Results 


is now 


should be 


is now 


should be 


is now 


should be 


X 

<D 

CO 


<D 

o> 

< 


Program 


C 

o 

Q. 

O 


Years Teaching 


Start 


30 


60 


40 


45 


60 


50 


50 


Best 


30 


30 


40 


44 


60 


71 




3AF COURSE ASSESSMENT - DATA SHEET OPTIMISATION OF TEACHING 


Part 2 For the Students 


# in class 


36 


# present | 20 






Skills 


Understanding 


Attitudes 


Enjoyment 


# 


is now 


should be 


is now 


should be 


is now 


should be 


12 


50 


50 


95 


98 


90 


95 


95 


2 


30 


3 


2 


6 


56% 


25 


35 


70 


60 


80 


80 


88 


55 


2 


25 


3 


2 


3 


57% 


35 


80 


40 


20 


60 


40 


50 


30 


2 


25 


3 


2 


3 


43% 


30 


40 • 


80 


70 


90 


70 


90 


75 


2 


21 


3 


3 


0 


71% 






























Part 3 


Summary Results | 


Lecturer's 


changes 






Alignment 


Mean post- 


Validation 






Scope 


Proportion 


Predictors 


course results 


Correlations 






Skills 


Underst 


Attitudes 


Skills 


Underst 


Attitudes 


MScope |MProp 


Enjoy | 


Acad 


Scope | 


Prop 






Whole 


Class 






n=20 


















Start 


100% 


13% 


-17% 


16% 


-2% 


-14% 


1.499 


0.380 


69.5% | 


| 63.0% 


-0.265 


-0.225 




Best 


0% 


10% 


19% 


-3% 


0% 


3% 


0.674 


0.154 






-0.307 


-0.594 






Option 


2 


History 




n=3 


















Start 


100% 


13% 


-17% 


16% 


-2% 


-14% 


1.861 


0.458 


60.0% | 


52.2% | 








Best 


0% 


33% 


10% 


-4% 


5% 


-1% 


1.221 


0.299 














Option 


4 


Modern Languages 


n=4 


















Start 


100% 


13% 


-17% 


16% 


-2% 


-14% 


1.200 


0.293 


72.5% | 


75.7% | 








Best 


13% 


3% 


10% 


1% 


-2% 


0% 


0.223 


0.065 













Figure 2 is in three parts Part 1 ‘For the Lecturer’, Part 2 ‘For the students’ and Part 3 ‘Summary Results’. 
In Part 1, the ‘Start’ row shows the seven numbers input from the Lecturer’s alignment form that was 
displayed in Figure 1. Part 2 ‘For the Students’ shows just a selection of four rows #12, #25, #35 and #30, 
from the alignment forms of all the students in the course. As well as the seven numbers from the students’ 
alignment forms, these rows have been extended to show variables of interest to the lecturer. The last 
column for the students shows their academic results. These are entered after the course and are used to 
validate the predictions from the in-course alignment indicators and to further validate the alignment 
theory. The ‘Summary Results’ section, Part 3, shows the calculated changes resulting from the lecturer’s 
start position. It is seen, for example, that the lecturer’s starting position for Skills of 30 for ‘is now’ and 60 
for ‘should be’ requires a 100% change, i.e. (60 - 30)/30= 100% 



Although not shown, the student changes were calculated in the same way and each compared with the 
lecturer’s changes. This was done for changes in both Scope and Proportion. Scope is calculated for each 
student as the total of the student/lecturer absolute differences in the raw numbers given for the three 
objectives. Proportion alignment for each student is calculated in a similar way but using proportional 
changes calculated from the six numbers. The Alignment indicators shown in the results section of Figure 

O 
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2 are the average of students’ scope alignments. From this start position the average scope Alignment for 
all students was 1.499. Students’ scope alignments were correlated with their academic results at -0.265 
Also their Proportions alignment was correlated with their reported enjoyment at -0.22. This means that 
the more aligned students and their lecturer were on Scope then the higher were the students’ academic 
results. Also, the more aligned students and lecturer were on Proportion then the more the students enjoyed 
the course. These results agree with the theory. 



We can do a similar analysis for any subgroup of interest. Figure 2 also shows this analysis for two separate 
option groups in Part 3 - for Option 2 ‘History’ students and for Option 4 ‘Modern language’ students. The 
Scope alignment and proportion alignment for these two sub-groups is given along with their respective 
mean enjoyment scores and mean academic results. It will be noticed that the Modern Language students 
were more aligned in both Scope and Proportion than were the History students. Correspondingly, we find 
that the mean academic results and enjoyment of the Modern Language students were higher than those of 
the History students. Although the sizes of the sub-groups were small, these comparative sub-group results 
are also in agreement with the alignment theory. 



Calculating changes that optimise teaching and learning 

We can use a simple linear programming algorithm to find those lecturer changes that would result in the 
best/minimum alignment. There are many options for this calculation that depend on the indicator of most 
interest and the lecturer’s freedom of choice. The first example in Figure 2 minimises the mean scope 
alignment for all students of 1 .499 and calculates the ‘Best’ changes that would give this minimised scope 
alignment. The resulting ‘Best’ changes are shown in Part 3 of Figure 2 as 0% for Skills, 10% for 
Understanding and 19% for Attitudes. Given the lecturer’s initial evaluation of the whole class as 30, 40 
and 60, these optimum changes imply that the lecturer should have been working towards 30 (no change), 
44 and 71. If the lecturer had done so then the students’ alignment scores would have changed and their 
correlations of academic attainment with the new scope alignments and proportion alignments would have 
improved to -0.307 and -0.59 respectively. This sensitivity analysis again demonstrates (i) the predictive 
validity of the alignment indicators, in that scope alignment and proportion alignment correlate with 
attainment and enjoyment, and (ii) the validity of the theory that the better the scope alignment then the 
higher is the academic attainment and the better the proportion alignment then the more the students enjoy 
the course. 



A similar sensitivity analysis was done for the History and Modern Language option groups by minimising 
their alignments to find the best changes for these two sub-groups. Figure 2 shows that the best alignments 
for these two groups are 1.221 and 0.223 respectively. The mean proportion alignments for these two 
option groups would then also have improved to 0.99 and 0.065 respectively. 



Traditional SETs are a ‘post mortem’ assessment, collected at the end of the course when it is too late to 
feedback to help the students who made the assessments. However, a lecturer does not have to wait until 
the course is over to optimise teaching using the alignment method. The data collected in-course can be 
processed by the same type of sensitivity analysis to calculate the optimum changes that should be made by 
the lecturer to maximize the students post-course academic attainment and/or course enjoyment. 
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Administrative Decision point assessment of quality teaching 

The lecturer may utilize the five-minute alignment form many times during the course to keep teaching on 
track. The administration uses it just once near the end of the course to calculate the final alignment score 
for that lecturer ’s quality of teaching. This results in a single decision point number that can be compared 
across the institution and used for promotion and tenure decisions. 



It will be noticed from Figure 2 that the minimum alignment that is possible for this group is 0.674. The 
minimum possible alignment for the History students was much higher at 1.221 than for the Modern 
Language students at 0.223. These minimum possible alignment scores illustrate the best teaching/learning 
that is possible with these sub-groups of students and reflects the fact that students are not all equally 
amenable to required educational changes in Skills, Understanding and Attitudes. To give the lecturer 
some protection from such intransigence the measure of quality teaching is taken as the actual alignment 
less this minimum possible alignment. In the example given the measure of quality teaching for the whole 
class is 1.499 and the best possible alignment for the whole class is 0.674. Hence, the quality teaching 
score for the whole class is 1 .499-0.674=0.825. This is shown in Table 1 . 



Table 1 : Quality Teaching (QT) scores for whole class and sub-groups 



Alignment scores 

Sub-Group Mean Scope Best Scope QT score 
Class 1.499 0.674 0.825 

Option 2 1.861 1.221 0.640 

Option 4 1.200 0.223 0.977 



It can also be noticed from Table 1 that, as often happens in class teaching, the quality of teaching was 
skewed more towards the needs of the more intransigent group. 



Discussion 

This paper has only touched on the classroom assessment use of the Alignment Method. It has not discussed 
the staff and course development aspects of the method or the many benefits the method offers for Quality 
Assurance compared to traditional SETs. 
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This alignment method of assessing teaching quality offers 10 main benefits: 

1 It identifies the quality of teaching experienced by each individual student. 

2 It can be used to identify groups of students that might be disadvantaged by the teaching. 

3 It offers detailed diagnostic reports to help the lecturer. 

4 It only takes 5 minutes to administer and the analysis is quick and low-cost. 

5 It can be given several times in-course resulting in optimum recommendations to keep teaching 
on track. 

6 It is sensitive to criteria considered important in different subject areas and by different Faculties 
and levels of students. 

7 It maintains lecturer/student trust and promotes higher quality teaching and higher quality learning. 

8 It protects academic freedom, is non-threatening and has built in protection for Faculty who 
teach intransigent students and difficult courses. 

9 It uses one standard form and gives one single decision-point number that can be used in 
institutional evaluations for comparing quality of teaching across the university e.g. for Quality 
Audits, teaching awards and for promotion and tenure decisions. 

10 Post-course correlations with academic standards evidence the reliability and validity of the 
instrument for each course and for subgroups of students taking each course on which it is used. 



This alignment method can be flexibly piloted at different levels within an institution - at the level of full 
institutional evaluation, at the level of staff and course development within Faculties, Schools or Departments 
and at the level of individual lecturers who are interested in improving the quality of their own teaching for 
their own students. Web-based Alignment software is now being developed that will enable lecturers and 
administrators from tertiary institutions worldwide to avail themselves of the benefits of using the alignment 
method in their own institutions. 
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